UPDATE: I cannot find a way to work around the Qt bug below; the fix below may patch the RV side, but inside Qt itself, when it calls the format() method it will still see 8-8-8-2 meaning it's initializing things and its internal FBOs in a way that is out of our control.

I have requested support from the Qt forums, I'm hoping we can escalate this one faster but it may not be possible, tbc.

https://forum.qt.io/topic/163843/qopenglwidget-with-rgba-10-10-10-2-isn-t-created-correctly

Linked issues

Autodesk: SG-41275
Community: #931

Summarize your change.

10-bit workflow stopped working.

At first people in the community noticed an inconsistency between the pixel type (10-10-10-2 bits) and the internal format (10-10-10-3 bits) in a few source files, so it was suggested to try to change the internal format to GL_RGB10_A2, which matches the expected 10-10-10-2 bits, but this did not address the issue.

Upon further investigation:

The problem is in Qt6's macOS Cocoa platform plugin in the QOpenGLWidget implementation.

File: qtbase/src/plugins/platforms/cocoa/qcocoaglcontext.mm
Lines: 267-271 (color channels), 278-279 (alpha channel)

When Qt receives the format description (10-10-10-2) or (8-8-8-8), it requests the OpenGL context appropriately. The OpenGL context creation works correctly in both cases.

But after it sets the pixel format, it queries the number of color bits, and in both cases (8-8-8-8 and 10-10-10-2) macOS returns 32.

The bug happens with what Qt does afterwards when it queries the number of color/alpha bits. there is no query for individual r,g,b bits, so it "assumes" that it can get this by dividing the total number of bits by 4.

// Line 267-271: qcocoaglcontext.mm - incorrect
int colorSize = pixelFormatAttribute(NSOpenGLPFAColorSize); // returns 32 in both cases
colorSize /= 4; // INCORRECT FOR 10-10-10-2

m_format.setRedBufferSize(colorSize);
m_format.setGreenBufferSize(colorSize);
m_format.setBlueBufferSize(colorSize);

// Line 278-279: Alpha is queried separately (this part is correct)
m_format.setAlphaBufferSize(pixelFormatAttribute(NSOpenGLPFAAlphaSize)); // Returns 8 or 2, so CORRECT IN ALL CASES

So, Qt returns the following formats:
For 8-8-8-8: Returns 32 → 32/4=8 → Reports 8-8-8-8 (correct by accident)
For 10-10-10-2: Returns 32 → 32/4=8 → Reports 8-8-8-2 (wrong)

I have verified that this is a Qt QSurfaceFormat reporting bug by calling glGetIntegerv() on the R, G, B channels, and indeed the values returned are 10, not 8. So we can confirm the OpenGL context is correctly configured as 10-10-10-2; it's only Qt's reporting of the created surface format that is wrong.

Therefore, this is an attempted workaround. It would have been nice if "QOpenGLWidget::format()" was virtual which would have increased my confidence in the fix, however it is not so I had to make another helper function and call that instead that queries the number of bits, doesn't rely blinly on the bits returned by QSurfaceFormat in the format() call.

Describe the reason for the change.

10-bit display workflow broken, an OpenGL error occurs.

Describe what you have tested and on which operating system.

Tested that the error dialog is gone, but unknown if 10-bit workflow is restored since I dont have 10-bit monitors or media that shows this as a big visual difference.

Waiting for a colleague on the adsk team (a member of the community) to pull this PR and give it a go.

Add a list of changes, and note any that might need special attention during the review.

If possible, provide screenshots.